Providing Actionable Uncertainties in Autonomous Vehicles

ABSTRACT

Systems and methods are provided for detecting objects of interest. A computing system can input sensor data to one or more first machine-learned models associated with detecting objects external to an autonomous vehicle. The computing system can obtain as an output of the first machine-learned models, data indicative of one or more detected objects. The computing system can determine data indicative of at least one uncertainty associated with the one or more detected objects and input the data indicative of the one or more detected objects and the data indicative of the at least one uncertainty to one or more second machine-learned models. The computing system can obtain as an output of the second machine-learned models, data indicative of at least one prediction associated with the one or more detected objects. The at least one prediction can be based at least in part on the detected objects and the uncertainty.

RELATED APPLICATION

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/691,124, titled “Providing ActionableUncertainties in Autonomous Vehicles,” filed on Jun. 28, 2018. U.S.Provisional Patent Application No. 62/691,124 is hereby incorporated byreference herein in its entirety.

FIELD

The present disclosure relates generally to improving the ability of anautonomous vehicle to navigate itself within an environment includingobjects external to the autonomous vehicle.

BACKGROUND

An autonomous vehicle is a vehicle that is capable of sensing itsenvironment and navigating without human input. In particular, anautonomous vehicle can observe its surrounding environment using avariety of sensors and can attempt to comprehend the environment byperforming various processing techniques on data collected by thesensors. Given knowledge of its surrounding environment, the autonomousvehicle can identify an appropriate motion path for navigating throughsuch surrounding environment.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or may be learned fromthe description, or may be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to acomputer-implemented method of detecting objects of interest thatincludes inputting, by a computing system comprising one or morecomputing devices, sensor data to one or more first machine-learnedmodels associated with detecting objects external to an autonomousvehicle. The method includes obtaining, by the computing system as anoutput of the one or more first machine-learned models, data indicativeof one or more detected objects external to the autonomous vehicle. Themethod includes determining, by the computing system, data indicative ofat least one uncertainty associated with the one or more detectedobjects. The method includes inputting, by the computing system, thedata indicative of the one or more detected objects and the dataindicative of the at least one uncertainty associated with the one ormore detected objects to one or more second machine-learned modelsconfigured to generate predictions in association with objects externalto the autonomous vehicle. The method includes obtaining, by thecomputing system as an output of the one or more second machine-learnedmodels, data indicative of at least one prediction associated with theone or more detected objects, the at least one prediction based at leastin part on the one or more detected objects and the at least oneuncertainty.

Another example aspect of the present disclosure is directed to acomputing system that includes one or more first machine-learned modelsassociated with detecting objects external to an autonomous vehiclebased at least in part on sensor data, one or more secondmachine-learned models configured to generate predictions in associationwith objects external to the autonomous vehicle, one or more processors,and one or more non-transitory computer-readable media that collectivelystore instructions that, when executed by the one or more processors,cause the computing system to perform operations. The operations includegenerating, as an output of the one or more first machine-learnedmodels, data indicative of one or more detected objects external to theautonomous vehicle, generating data indicative of at least oneuncertainty associated with the one or more detected objects, inputtingthe data indicative of the one or more detected objects and the dataindicative of the at least one uncertainty associated with the one ormore detected objects to the one or more second machine-learned models,and generating, as an output of the one or more second machine-learnedmodels, data indicative of at least one prediction associated with theone or more detected objects. The at least one prediction is based atleast in part on the one or more detected objects and the at least oneuncertainty.

Yet another example aspect of the present disclosure is directed to anautonomous vehicle that includes one or more processors, and one or morenon-transitory computer-readable media that collectively storeinstructions that, when executed by the one or more processors, causethe one or more processors to perform operations. The operations includeinputting sensor data to one or more first machine-learned modelsassociated with detecting objects external to an autonomous vehicle,obtaining, as an output of the one or more first machine-learned models,data indicative of one or more detected objects external to theautonomous vehicle, determining data indicative of at least oneuncertainty associated with the one or more detected objects, inputtingthe data indicative of the one or more detected objects and the dataindicative of the at least one uncertainty associated with the one ormore detected objects to one or more second machine-learned modelsconfigured to generate predictions in association with objects externalto the autonomous vehicle, and obtaining, as an output of the one ormore second machine-learned models, data indicative of at least oneprediction associated with the one or more detected objects, the atleast one prediction based at least in part on the one or more detectedobjects and the at least one uncertainty.

These and other features, aspects and advantages of various embodimentswill become better understood with reference to the followingdescription and appended claims. The accompanying drawings, which areincorporated in and constitute a part of this specification, illustrateembodiments of the present disclosure and, together with thedescription, serve to explain the related principles.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art are set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts a block diagram of an example autonomous vehicleaccording to example embodiments of the present disclosure;

FIG. 2 depicts a block diagram of an example perception system accordingto example embodiments of the present disclosure;

FIG. 3 depicts block diagram of an example perception system andprediction system, including a probability system according to exampleembodiments of the present disclosure;

FIG. 4 depicts a flowchart diagram of an example process of generatingdata indicative of uncertainty associated with object detections andproviding the data indicative of uncertainty downstream for objectprediction and/or motion planning according to example embodiments ofthe present disclosure;

FIG. 5 depicts block diagram of an example prediction system and motionplanning system, including a probability system according to exampleembodiments of the present disclosure;

FIG. 6 depicts a flowchart diagram of an example process of generatingmotion plans for an autonomous vehicle based on data indicative ofuncertainty according to example embodiments of the present disclosure;

FIG. 7 depicts a block diagram of an example perception system andprediction system, including probability generators integrated withinmachine-learned models of the perception system according to exampleembodiments of the present disclosure;

FIG. 8 depicts a block diagram of an example perception system includingone or more fusing layers in which a probability generator is integratedaccording to example embodiments of the present disclosure;

FIG. 9 is a flowchart diagram of an example process of training amachine-learned model to generate true estimates of class probabilitiesfor detected objects in example embodiments of the present disclosure;and

FIG. 10 depicts a block diagram of an example computing system accordingto example embodiments of the present disclosure.

DETAILED DESCRIPTION

Reference now will be made in detail to embodiments, one or moreexample(s) of which are illustrated in the drawings. Each example isprovided by way of explanation of the embodiments, not limitation of thepresent disclosure. In fact, it will be apparent to those skilled in theart that various modifications and variations can be made to theembodiments without departing from the scope or spirit of the presentdisclosure. For instance, features illustrated or described as part ofone embodiment can be used with another embodiment to yield a stillfurther embodiment. Thus, it is intended that aspects of the presentdisclosure cover such modifications and variations.

Generally, the present disclosure is directed to systems and methodsthat generate and share uncertainty data between components of anautonomous vehicle computing system to improve object prediction andmotion planning for the autonomous vehicle. In some implementations, forexample, a perception system of the vehicle computing system maygenerate data indicative of one or more objects detected based at leastin part on sensor data from a sensor system of the autonomous vehicle.The vehicle computing system may additionally generate uncertainty dataincluding data indicative of an uncertainty associated with the one ormore detected objects. The data indicative of the one or more detectedobjects and the data indicative of the uncertainty associated with theone or more detected objects can be provided to additional components ofthe vehicle computing system, such as a prediction system and/or motionplanning system. The additional components can utilize the uncertaintyassociated with the detected objects to improve object prediction and/ormotion planning processes for the autonomous vehicle.

More particularly, in some implementations, the vehicle computing systemcan generate data indicative of one or more object detections using oneor more machine-learned models associated with a perception system. Thevehicle computing system may determine an uncertainty associated withthe object detections provided by the machine-learned model(s)configured for object detection, tracking, and/or classification. Dataindicative of a determined uncertainty (also referred to as anuncertainty data) associated with a detected object can be provided toone or more additional machine-learned models that are configured forobject prediction and/or motion planning for the autonomous vehicle. Theuncertainty associated with a detected object can be used with dataindicative of the detected object to generate an object prediction, suchas an expected trajectory of the object, and/or a motion plan for theautonomous vehicle.

An autonomous vehicle can be a ground-based autonomous vehicle (e.g.,car, truck, bus, etc.) or other type of vehicle. The autonomous vehiclecan include a computing system that assists in controlling theautonomous vehicle. In some implementations, the autonomous vehiclecomputing system can include a perception system, a prediction system, amotion planning system, and an uncertainty system that cooperate toperceive the surrounding environment of the autonomous vehicle anddetermine one or more motion plans for controlling the motion of theautonomous vehicle accordingly. The autonomous vehicle computing systemcan include one or more processors as well as one or more non-transitorycomputer-readable media that collectively store instructions that, whenexecuted by the one or more processors, cause the autonomous vehiclecomputing system to perform various operations as described herein.

In some implementations, a perception system of a vehicle computingsystem may include one or more first machine-learned models associatedwith detecting objects external to an autonomous vehicle. By way ofexample, the one or more first machine-learned models may include objectdetection models included as part of a segmentation or detectioncomponent of the perception system, object tracking models included aspart of a tracking component of the perception system, and/orclassification models included as part of a classification component ofthe perception system. An object detection model may generate dataindicative of a detected object, also referred to as object detectiondata. For example an object detection model may obtain sensor data fromone or more sensors of a sensor system and provide data indicating thatan object has been detected. For instance, an object detection model maygenerate, based on the sensor data, object detection data indicating thepresence or existence of a vehicle, pedestrian, bicycle, or other objecttype in an environment external to the autonomous vehicle. An objecttracking model may generate data indicative of a state of a detectedobject, also referred to as state data. For example, an object trackingmodel may generate data relating to an object's position, velocity,acceleration, heading, or other information relative to an object'scurrent state. In some examples, an object tracking model may combineinformation from multiple cycles or frames of sensor data as part ofdetermining object state information. A classification model maygenerate data indicative of a classification of a detected object, alsoreferred to as classification data. For example, a classification modelmay generate data indicating to which of a plurality of classes adetected object belongs. Example classes of objects for which aclassification model may provide a classification may include vehicles,bicycles, pedestrians, signs, or any other type of object for which aclassification is desired.

A vehicle computing system in accordance with embodiments of thedisclosed technology may generate data indicative of an uncertaintyassociated with an output of any of the models included as part of aperception system. For example, the vehicle computing system maygenerate detection uncertainty data including data indicative of anuncertainty associated with an object detection provided by an objectiondetection model. The system may generate state uncertainty dataincluding data indicative of an uncertainty associated with an objectstate provided by an object tracking model. The system may generateclassification uncertainty data including data indicative of anuncertainty associated with an object classification provided by aclassification model.

In some examples, the data indicative of the uncertainty associated withone or more detected objects may be provided to one or more secondmachine-learned models of a prediction system, in addition to the dataindicative of the one or more detected objects. The one or more secondmachine-learned models of the prediction system can then determine oneor more object predictions based at least in part on the one or moredetected objects as well as the uncertainty associated with the one ormore detected objects. By way of example, the one or more secondmachine-learned models may determine a predicted trajectory of anobject, a predicted position of an object, a predicted velocity of anobject, a predicted acceleration of an object, a predicted heading of anobject, or other information pertaining to a predicted state of anobject. The prediction system may utilize the uncertainty associatedwith an object detection as part of generating a prediction using theone or more second machine-learned models.

In some implementations, data indicative of uncertainty associated withdetected objects can be provided to one or more third machine-learnedmodels of the motion planning system. The data indicative of uncertaintymay be provided to the motion planning system, in addition to the dataindicative of the detected objects, to aid in motion planning based ondetected objects in some examples. The one or more third machine-learnedmodels of the motion planning system can generate one or more motionplans based at least in part on the one or more detected objects, aswell as the uncertainty associated with the one or more detectedobjects.

According to some example embodiments, an uncertainty system mayadditionally or alternatively determine an uncertainty associated withobject predictions. Data indicative of an uncertainty associated with anobject prediction can be provided to the motion planning system, inaddition to the data indicative of the object prediction. By way ofexample, the one or more second machine-learned models may generate anobject prediction, such as a predicted trajectory of an object. Theuncertainty system can determine an uncertainty associated with theobject prediction. Data indicative of the uncertainty associated withthe object prediction can then be provided to the motion planning systemto aid in motion planning processes. For instance, the prediction systemmay calculate two or more predicted trajectories for an object andprovide probability distributions for each of the predicted trajectoriesto the motion planning system. In this manner, the motion planningsystem may generate one or more motion plans based on both of thepredicted trajectories, as well as the probability distribution for eachof those trajectories. In some examples, the motion planning system onlyreceives uncertainty data associated with the prediction system, such asa probability distribution associated with a predicted objecttrajectory. In other examples, however, the motion planning system canadditionally or alternatively receive uncertainty data associated withthe perception system, such as a probability distribution associatedwith an object detection or classification.

According to example embodiments, the uncertainty of an object may bedetermined based at least in part on an object state, an object history,a local context associated with an object, and/or a global contextassociated with an object. The object state may include variousparameters such as position, heading, shape, velocity, acceleration,etc. associated with an object. A tracking component of the perceptionsystem may provide an output including state and/or uncertainty dataassociated with the state of an object. The state information can beused with history data associated with object detections to determine anuncertainty. History information such as one or more previous objectdetections may be used to determine uncertainty associated with anobject detection. Thus, the length of time or number of cycles duringwhich an object has been detected and/or tracked can be used as part ofa current object detection and uncertainty determination. For example,the system may use a classification or state associated with an objectfrom a previous cycle as part of determining the classification or stateof the object for a current cycle. More particularly, the uncertaintysystem may be configured to increase the uncertainty associated with anobject if its classification or state changes between cycles, as thismay be indicative of a lack of certainty in the detection or prediction.Additionally, a history of the uncertainty associated with an object canbe used as part of determining a current object detection anduncertainty associated with the detection.

In some examples, a predetermined number of previous states of an objectcan be stored and used to determine an uncertainty associated with adetected object. The most recent states of an object can be passed tothe prediction system such that history features can be calculated foran object. Additionally, a predetermined number of future states can beforward simulated and passed as a placeholder to be used for predictingtrajectories using instance flow. The past state determinations andfuture predictions can be used as part of determining and/or modifyingan uncertainty associated with an object. Uncertainties or previousstate information can be modified to be more in-line with a fulltrajectory of an object. The use of past history features may improveprediction, as an analysis of a trajectory may allow a determination ofto what degree an object detection should be trusted by downstreamcomponents.

A local context associated with a detected object may also be used aspart of determining uncertainty data associated with the object. Forexample, the system may receive sensor data from multiple sensors andgenerate object detection data based on a perception pipeline or otherdetection system associated with each of the sensors. According toexample embodiments, uncertainty data may be generated for one or moreobject detections based on the outputs of multiple perception pipelines.For instance, the system may include a fusing system to combine theoutputs of multiple perception pipelines. The system may provide anuncertainty for the one or more object detections based at least in parton the outputs of the various pipelines. For instance, the system mayuse a concurrence between the output of multiple pipelines to increasethe uncertainty associated with an object detection or a disagreementbetween outputs to decrease the uncertainty.

Additionally, a global context associated with a detected object may beused as part of determining uncertainty associated with an object. Thesystem may use an object's position or locality with respect to thevehicle and/or a number of points in sensor data corresponding to theobject to determine an uncertainty associated with the object. Forexample, an object detection associated with a small number of points(e.g., one) at far range from the vehicle may be considered to betrustworthy. Because an object at far range can be expected to only havea small number of sensor data points, a low uncertainty can be assignedto the object detection. By contrast, an object detection associatedwith a small number of points at close range to the vehicle may beconsidered untrustworthy. Because an object at close range can beexpected to have a large number of sensor data points, a highuncertainty can be assigned to the object detection. Map information canadditionally be used to provide global context as part of generating anuncertainty in association with an object detection. Map data can beused to determine whether an object detection is consistent with aglobal context of a scene or environment external to the vehicle. Thus,map data can be used independently of detection to provide an assessmentof the consistency of a detection with the environment. Map informationmay be used with statistics associated with objects in a scene to gatherinformation (e.g., using a graph structure) in order to provideuncertainty information associated with the objects.

In some example implementations, a vehicle computing system may includea separate uncertainty system configured to determine uncertainty dataassociated with object detections and/or object predictions. Forexample, the vehicle computing system may include an uncertainty systemthat receives one or more outputs of one or more machine-learned modelsof the perception system and/or prediction system. For instance, dataindicative of an object detection, object state, and/or objectclassification may be provided from the appropriate machine-learnedmodel to the uncertainty system. The uncertainty system may determine anuncertainty associated with any one of the models, such as anuncertainty associated with whether the object was actually detected, anuncertainty associated with the state of the object, and/or anuncertainty associated with the classification of the object. Theuncertainty system may then provide data indicative of uncertaintyassociated with detected objects to the prediction system. The dataindicative of the uncertainty associated with the detected objects canbe provided in addition to the data indicative of the detected objects.In some examples, the external uncertainty system may receive one ormore outputs of one or more machine-learned models of the predictionsystem. The uncertainty system can determine an uncertainty associatedwith one or more object predictions and provide the uncertainty data tothe motion planning system in addition to the data indicative of theobject prediction.

In other example embodiments, an uncertainty system may be implementedwithin one or more machine-learned models configured for objectdetection or object prediction. For example, a machine-learned modelassociated with object detection may be configured to generate dataindicative of the uncertainty associated with a detected object, inaddition to generating data indicative of the detected object.Similarly, machine-learned models associated with object tracking andobject classification may be configured to generate data indicative ofan uncertainty associated with an object state and objectclassification, respectively. Likewise, machine-learned modelsassociated with object prediction can be configured to generateuncertainty associated with object predictions. In some examples, anuncertainty system may be implemented partially within one or moremachine-learned models and partially external to the one or moremachine-learned models.

An uncertainty system can be implemented at least partially within oneor more fusing layers of a perception system and/or prediction system insome example implementations. One or more fusing layers may be providedwithin a perception system, for example, to fuse multiple perceptionoutputs associated with different sensors of the autonomous vehicle. Forinstance, the sensor data associated with a first sensor may beprocessed using a first perception pipeline including a first set ofmachine-learned models for object detection, object tracking, and/orobject classification. The sensor data associated with a second sensormay be processed using a second perception pipeline including adifferent set of machine-learned models for object detection, objecttracking, and/or object classification. The one or more fusing layerscan receive the outputs of the different perception pipelines anddetermine a final object detection, final object state, and/or finalobject classification from the outputs of the different perceptionpipelines. According to some embodiments, the probability system maydetermine an uncertainty associated with a detected object based onoutputs of different perception pipelines associated with differentsensors of the autonomous vehicle.

An uncertainty system in accordance with example embodiments maydetermine a probability distribution for different outputs associatedwith different perception pipelines in some implementations. As aspecific example, a first perception pipeline may generate a firstobject classification for a detected object while a second perceptionpipeline may generate a second classification for the same detectedobject. The uncertainty system may generate a probability distributionassociated with the different object classifications and provide thatprobability distribution to the prediction system in addition to both ofthe object classifications generated by the different perceptionpipelines.

In accordance with some embodiments, one or more machine-learned modelsmay be utilized to generate data indicative of uncertainty associatedwith object detections and/or object predictions. As earlier described,a machine-learned model configured for object detection, tracking,and/or classification may be additionally configured to generate dataindicative of uncertainty associated with its outputs. In some examples,one or more additional machine-learned models external to themachine-learned models configured for object detection, tracking, and/orclassification may be used to generate data indicative of uncertainty.In either case, a machine-learned model in accordance with exampleembodiments can be trained to generate a true probability distributionfor its output based on a distribution of inputs. For example, themachine-learned model may be trained to generate multiple outputs and anactual probability or uncertainty associated with each output, ratherthan be trained to generate a single output with a highest level ofcertainty.

More particularly, in some examples, a model configured to generateuncertainty data associated with object detections and/or objectpredictions may provide a continuous, calibrated uncertainty inassociation with a perception system or prediction system output. Such amodel can be trained by analyzing the joint probability of all outputsgiven a set of input data. As a specific example, a Bayesian binaryclassifier may be used in some examples, also referred to as a Gaussiannaïve Bayes. Using such a model, underlying data can be analyzed toproduce a true estimate of probabilities given a distribution of theinputs seen during training of the machine-learned model. In thismanner, a continuous calibrated uncertainty can be provided whereby theprobability is continuous between a value of zero and one and is not adiscrete value of zero or one as may be the case with the outputs ofmany classifiers. Uncertainty can be calibrated such that it isrepresentative of a true distribution. A true probability distributionhaving a calibrated uncertainty may provide actionable levels ofuncertainty. As a contrast, many discriminative models are trained toprovide classifications for a most confident determination which isclose to zero or one.

According to one example aspect of the present disclosure, a computingsystem comprising one or more computing devices is provided. Thecomputing system can be configured to perform a computer-implementedmethod of detecting objects. The method can include inputting, by thecomputing system, sensor data to one or more machine-learned modelsassociated with detecting objects external to an autonomous vehiclebased at least in part on sensor data. The method can include obtaining,by the computing system, as an output of the one or more machine-learnedmodels, data indicative of one or more detected objects external to theautonomous vehicle. The method can include generating, by the computingsystem, data indicative of at least one uncertainty associated with theone or more detected objects. The method can include inputting the dataindicative of the one or more detected objects and the data indicativeof the at least one uncertainty associated with the one or more detectedobjects to one or more second machine-learned models configured togenerate predictions in association with objects external to theautonomous vehicle. The method can include obtaining, by the computingsystem as an output of the one or more second machine-learned models,data indicative of at least one prediction associated with the one ormore detected objects. The at least one prediction is based at least inpart on the one or more detected objects and the data indicative of theat least one uncertainty.

According to one example aspect of the present disclosure, the computingsystem can additionally input the data indicative of the at least oneuncertainty associated with one or more detected objects and the dataindicative of the at least one prediction associated with one or moredetected objects to one or more third machine-learned models. Thecomputing system can obtain as an output of the one or more thirdmachine-learned models, data indicative of at least one motion planbased on one or more detected objects. The computing system can controlthe autonomous vehicle based at least in part on the at least one motionplan. In some examples, the data indicative of the at least one motionplan can be generated based at least in part on the data indicative ofthe at least one uncertainty associated with the one or more detectedobjects and the data indicative of the at least one predictionassociated with the one or more detected objects.

In some implementations, the computing system can additionally generatedata indicative of at least one uncertainty associated with a predictionassociated with one or more detected objects. The at least oneuncertainty associated with the prediction can be input by the computingsystem to the one or more third machine-learned models along with dataindicative of the at least one prediction. The one or more thirdmachine-learned models can generate one or more motion plans based atleast in part on the data indicative of the uncertainty associated withthe prediction.

In some implementations, the computing system can generate dataindicative of at least one uncertainty using an uncertainty system thatis external to one or more machine-learned models for which theuncertainty system is generating the at least one uncertainty.Additionally and/or alternatively, the computing system can generatedata indicative of at least one uncertainty using an uncertainty systemthat includes one or more components integrated within the one or moremachine-learned models for which the uncertainty system is generatingthe at least one uncertainty.

The systems and methods of the present disclosure provide a number oftechnical effects and benefits, particularly in the areas of autonomousvehicles and computing technology. An autonomous vehicle in accordancewith embodiments of the disclosed technology may generate dataindicative of uncertainty associated with detected objects in theenvironment external to the autonomous vehicle and/or predictions withrespect to the detected objects. The various data indicative ofuncertainty may be used by a vehicle computing system to improveautonomous vehicle operations. By way of example, the autonomous vehiclemay utilize data indicative of uncertainty associated with detectedobjects and/or object predictions to generate improved predictionsand/or motion plans. As a specific example, the use of data indicativeof uncertainty may allow the system to avoid rapid changes betweenobject detections and/or object classifications. The introduction ofdata indicative of uncertainty may allow the system to more accuratelydetermine when to change an object's detection or classification. As aspecific example, the utilization of uncertainty may reduce or otherwisealleviate hard switching between motion plans which may result in hardbraking or hard turns of the vehicle in some instances. The utilizationof actionable uncertainty may be contrasted with techniques that utilizea single best detection or classification without data indicative ofuncertainty. In such systems, a detection or classification of adetected object may change from cycle to cycle, resulting in updated ornew motion plans which may cause hard braking or turning of the vehicle.

The use of uncertainty provides additional actionable information tovarious systems of the autonomous vehicle to improve general operationof the autonomous vehicle. More specifically, the use of uncertaintydata allows multiple possible detections, states, and/or classificationsto be passed between components of the vehicle computing system. Thevarious components can then use the uncertainty data to determine how toprocess multiple detections, changing classifications, etc. Such atechnique may provide improved object predictions and motion planningwhen compared with systems that choose a single best detection orprediction, and only pass final determined information to othercomponents of the vehicle computing system. Rather than discardinformation as relating to a less certain output, the system can passthe data with additional probability information to improve downstreamoperations. In some examples, uncertainty data may be used to adjust thespeed and/or defensive driving behavior of an autonomous vehicle. Forexample, in response to determining that a detected object is associatedwith a high level uncertainty, the autonomous vehicle may initiate amotion plan to cause the autonomous vehicle to operate at slower speedsuntil the object's uncertainty is reduced. As yet another example, aregion or buffer around a detected object in which the autonomousvehicle will not encroach may be increased in response to higher levelsof uncertainty associated with the object.

The systems and methods described herein may provide a particulartechnical benefit to vehicle computing systems of autonomous vehicles.In particular, a vehicle computing system can generate data indicativeof uncertainty with respect to an object detected using one or moremachine-learned models. The data indicative of uncertainty can be passedwith object detection data to one or more machine-learned modelsconfigured for object prediction. The machine-learned models can use thedata indicative of uncertainty to generate an improved prediction forthe object. Additionally or alternatively, the data indicative ofuncertainty associated with the object detection data can be provided toone or more machine-learned models for motion planning. Themachine-learned models configured for motion planning can use the dataindicative of uncertainty to generate improved motion plans for theautonomous vehicle. The use of uncertainty data may further improveefficiencies in vehicle computing systems for autonomous vehicles. Forexample, multiple classifications for the same object can be associatedwith a probability distribution that is passed to a prediction or motionplanning system. The downstream system can then use the additionalprobability information to generate improved predictions or motionplans, rather than discard the already generated data.

With reference now to the Figures, example embodiments of the presentdisclosure will be discussed in further detail. FIG. 1 depicts a blockdiagram of an example autonomous vehicle 10 according to exampleembodiments of the present disclosure. The autonomous vehicle 10 iscapable of sensing its environment and navigating without human input.The autonomous vehicle 10 can be a ground-based autonomous vehicle(e.g., car, truck, bus, etc.), an air-based autonomous vehicle (e.g.,airplane, drone, helicopter, or other aircraft), or other types ofvehicles (e.g., watercraft, rail-based vehicles, etc.).

The autonomous vehicle 10 includes one or more sensors 101, a vehiclecomputing system 102, and one or more vehicle controls 107. The vehiclecomputing system 102 can assist in controlling the autonomous vehicle10. In particular, the vehicle computing system 102 can receive sensordata from the one or more sensors 101, attempt to comprehend thesurrounding environment by performing various processing techniques ondata collected by the sensors 101, and generate an appropriate motionpath through such surrounding environment. The vehicle computing system102 can control the one or more vehicle controls 107 to operate theautonomous vehicle 10 according to the motion path.

The vehicle computing system 102 includes a computing device 110including one or more processors 112 and a memory 114. The one or moreprocessors 112 can be any suitable processing device (e.g., a processorcore, a microprocessor, an ASIC, a FPGA, a controller, amicrocontroller, etc.) and can be one processor or a plurality ofprocessors that are operatively connected. The memory 114 can includeone or more non-transitory computer-readable storage mediums, such asRAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., andcombinations thereof. The memory 114 can store data 116 and instructions118 which are executed by the processor 112 to cause vehicle computingsystem 102 to perform operations.

As illustrated in FIG. 1 , the vehicle computing system 102 can includea perception system 103, a prediction system 104, and a motion planningsystem 105 that cooperate to perceive the surrounding environment of theautonomous vehicle 10 and determine a motion plan for controlling themotion of the autonomous vehicle 10 accordingly.

In particular, in some implementations, the perception system 103 canreceive sensor data from the one or more sensors 101 that are coupled toor otherwise included within the autonomous vehicle 10. As examples, theone or more sensors 101 can include a Light Detection and Ranging(LIDAR) system, a Radio Detection and Ranging (RADAR) system, one ormore cameras (e.g., visible spectrum cameras, infrared cameras, etc.),and/or other sensors. The sensor data can include information thatdescribes the location of objects within the surrounding environment ofthe autonomous vehicle 10.

As one example, for a LIDAR system, the sensor data can include thelocation (e.g., in three-dimensional space relative to the LIDAR system)of a number of points that correspond to objects that have reflected aranging laser. For example, a LIDAR system can measure distances bymeasuring the Time of Flight (TOF) that it takes a short laser pulse totravel from the sensor to an object and back, calculating the distancefrom the known speed of light.

As another example, for a RADAR system, the sensor data can include thelocation (e.g., in three-dimensional space relative to the RADAR system)of a number of points that correspond to objects that have reflected aranging radio wave. For example, radio waves (e.g., pulsed orcontinuous) transmitted by the RADAR system can reflect off an objectand return to a receiver of the RADAR system, giving information aboutthe object's location and speed. Thus, a RADAR system can provide usefulinformation about the current speed of an object.

As yet another example, for one or more cameras, various processingtechniques (e.g., range imaging techniques such as, for example,structure from motion, structured light, stereo triangulation, and/orother techniques) can be performed to identify the location (e.g., inthree-dimensional space relative to the one or more cameras) of a numberof points that correspond to objects that are depicted in imagerycaptured by the one or more cameras. Other sensor systems can identifythe location of points that correspond to objects as well.

As another example, the one or more sensors 101 can include apositioning system. The positioning system can determine a currentposition of the autonomous vehicle 10. The positioning system can be anydevice or circuitry for analyzing the position of the autonomous vehicle10. For example, the positioning system can determine position by usingone or more of inertial sensors, a satellite positioning system, basedon IP address, by using triangulation and/or proximity to network accesspoints or other network components (e.g., cellular towers, WiFi accesspoints, etc.) and/or other suitable techniques. The position of theautonomous vehicle 10 can be used by various systems of the vehiclecomputing system 102.

Thus, the one or more sensors 101 can be used to collect sensor datathat includes information that describes the location (e.g., inthree-dimensional space relative to the autonomous vehicle 10) of pointsthat correspond to objects within the surrounding environment of theautonomous vehicle 10.

In addition to the sensor data, the perception system 103 can retrieveor otherwise obtain map data 126 that provides detailed informationabout the surrounding environment of the autonomous vehicle 10. The mapdata 126 can provide information regarding: the identity and location ofdifferent travelways (e.g., roadways), road segments, buildings, orother items or objects (e.g., lampposts, crosswalks, curbing, etc.); thelocation and directions of traffic lanes (e.g., the location anddirection of a parking lane, a turning lane, a bicycle lane, or otherlanes within a particular roadway or other travelway); traffic controldata (e.g., the location and instructions of signage, traffic lights, orother traffic control devices); and/or any other map data that providesinformation that assists the vehicle computing system 102 incomprehending and perceiving its surrounding environment and itsrelationship thereto.

The perception system 103 can identify one or more objects that areproximate to the autonomous vehicle 10 based on sensor data receivedfrom the one or more sensors 101 and/or the map data 126. In particular,in some implementations, the perception system 103 can determine, foreach object, state data that describes a current state of such object asdescribed. As examples, the state data for each object can describe anestimate of the object's: current location (also referred to asposition); current speed (also referred to as velocity); currentacceleration; current heading; current orientation; size/footprint(e.g., as represented by a bounding shape such as a bounding polygon orpolyhedron); class (e.g., vehicle versus pedestrian versus bicycleversus other); yaw rate; and/or other state information.

In some implementations, the perception system 103 can determine statedata for each object over a number of iterations. In particular, theperception system 103 can update the state data for each object at eachiteration. Thus, the perception system 103 can detect and track objects(e.g., vehicles) that are proximate to the autonomous vehicle 10 overtime.

The prediction system 104 can receive the state data from the perceptionsystem 103 and predict one or more future locations for each objectbased on such state data. For example, the prediction system 104 canpredict where each object will be located within the next 5 seconds, 10seconds, 20 seconds, etc. As one example, an object can be predicted toadhere to its current trajectory according to its current speed. Asanother example, other, more sophisticated prediction techniques ormodeling can be used.

The motion planning system 105 can determine one or more motion plansfor the autonomous vehicle 10 based at least in part on the predictedone or more future locations for the object and/or the state data forthe object provided by the perception system 103. Stated differently,given information about the current locations of objects and/orpredicted future locations of proximate objects, the motion planningsystem 105 can determine a motion plan for the autonomous vehicle 10that best navigates the autonomous vehicle 10 relative to the objects attheir current and/or future locations.

As one example, in some implementations, the motion planning system 105can evaluate one or more cost functions for each of one or morecandidate motion plans for the autonomous vehicle 10. For example, thecost function(s) can describe a cost (e.g., over time) of adhering to aparticular candidate motion plan and/or describe a reward for adheringto the particular candidate motion plan. For example, the reward can beof opposite sign to the cost.

The motion planning system 105 can provide the optimal motion plan to avehicle controller 106 that controls one or more vehicle controls 107(e.g., actuators or other devices that control gas flow, steering,braking, etc.) to execute the optimal motion plan. The vehiclecontroller can generate one or more vehicle control signals for theautonomous vehicle based at least in part on an output of the motionplanning system.

Each of the perception system 103, the prediction system 104, the motionplanning system 105, and the vehicle controller 106 can include computerlogic utilized to provide desired functionality. In someimplementations, each of the perception system 103, the predictionsystem 104, the motion planning system 105, and the vehicle controller106 can be implemented in hardware, firmware, and/or softwarecontrolling a general purpose processor. For example, in someimplementations, each of the perception system 103, the predictionsystem 104, the motion planning system 105, and the vehicle controller106 includes program files stored on a storage device, loaded into amemory and executed by one or more processors. In other implementations,each of the perception system 103, the prediction system 104, the motionplanning system 105, and the vehicle controller 106 includes one or moresets of computer-executable instructions that are stored in a tangiblecomputer-readable storage medium such as RAM hard disk or optical ormagnetic media.

In various implementations, one or more of the perception system 103,the prediction system 104, and/or the motion planning system 105 caninclude or otherwise leverage one or more machine-learned models suchas, for example convolutional neural networks.

FIG. 2 depicts a block diagram of an example perception system 103according to example embodiments of the present disclosure. As discussedin regard to FIG. 1 , a vehicle computing system 102 can include aperception system 103 that can identify one or more objects that areproximate to an autonomous vehicle 10. In some embodiments, theperception system 103 can include segmentation component 206, objectassociations component 208, tracking component 210, tracked objectscomponent 212, and classification component 214. The perception system103 can receive sensor data 202 (e.g., from one or more sensor(s) 101 ofthe autonomous vehicle 10) and map data 204 as input. The perceptionsystem 103 can use the sensor data 202 and the map data 204 indetermining objects within the surrounding environment of the autonomousvehicle 10. In some embodiments, the perception system 103 iterativelyprocesses the sensor data 202 to detect, track, and classify objectsidentified within the sensor data 202. In some examples, the map data204 can help localize the sensor data to positional locations within amap data or other reference system.

Within the perception system 103, the segmentation component 206 canprocess the received sensor data 202 and map data 204 to determinepotential objects within the surrounding environment, for example usingone or more object detection systems. The object associations component208 can receive data about the determined objects and analyze priorobject instance data to determine a most likely association of eachdetermined object with a prior object instance, or in some cases,determine if the potential object is a new object instance. The trackingcomponent 210 can determine the current state of each object instance,for example, in terms of its current position, velocity, acceleration,heading, orientation, uncertainties, and/or the like. The trackedobjects component 212 can receive data regarding the object instancesand their associated state data and determine object instances to betracked by the perception system 103. The classification component 214can receive the data from tracked objects component 212 and classifyeach of the object instances. For example, classification component 214can classify a tracked object as an object from a predetermined set ofobjects (e.g., a vehicle, bicycle, pedestrian, etc.). The perceptionsystem 103 can provide the object and state data for use by variousother systems within the vehicle computing system 102, such as theprediction system 104 of FIG. 1 .

FIG. 3 is a block diagram depicting an example of a vehicle computingsystem including an uncertainty system in accordance with exampleembodiments of the present disclosure. FIG. 3 depicts a subset ofcomponents of an example vehicle computing, namely a perception system103 and prediction system 104. Perception system 103 includes anuncertainty system 302 in accordance with example embodiments of thedisclosed technology.

Uncertainty system 302 is configured to generate uncertainty data inassociation with one or more outputs of perception system 103.Uncertainty system 302 receives data indicative of one or more detectedobjects. The data indicative of one or more objects may include objectdetection data from segmentation component 206, state data from trackingcomponent 210, and/or classification data from classification component214.

Object detection data generated by segmentation component 206 mayinclude data indicative of an object detection. The data may indicatethe presence of one or more detected objects in an environment externalto the autonomous vehicle. In some examples, the data may be based on anumber of points in sensor data indicating the presence of the object.By way of example, the object detection data may indicate the presenceor existence of a vehicle, pedestrian, bicycle, or other object type. Insome examples, segmentation component 206 may include one or more objectdetectors 304 configured to generate object detection data. Each objectdetector 304 may include one or more machine-learned models configuredto generate object detection data based on input sensor data 202.

State data generated by tracking component 210 may include dataindicative of the state of one or more detected objects. The data mayindicate state information associated with the one or more detectedobjects. By way of example, state data may include data relating to anobject's position, velocity, acceleration, heading, orientation, size orother information relative to an object's current state. In someexamples, tracking component 210 may include one or more tracking modelsconfigured to generate state data for detected objects. Each trackingmodel may include one or more machine-learned models configured togenerate state data for one or more object detections. The trackingmodels may combine information from multiple cycles or frames of sensordata as part of determining object state information.

Classification data generated by classification component 214 mayinclude data indicative of a classification of one or more detectedobjects. By way of example, classification data may include dataindicating whether a detected object is a vehicle, pedestrian, bicycle,or other classification of object encountered in the environment of anautonomous vehicle. In some examples, classification component 214 mayinclude one or more classification models configured to generateclassification data for detected objects. Each classification model mayinclude one or more machine-learned models configured to generateclassification data for one or more object detections.

Uncertainty system 302 is configured to generate uncertainty data inassociation with each of the received outputs of the perception system.Uncertainty system 302 can generate detection uncertainty data inassociation with object detection data generated by segmentationcomponent 206. Uncertainty system 302 can generate state uncertaintydata in association with state data generated by tracking component 210.Uncertainty system 302 can generate classification uncertainty data inassociation with classification data generated by classificationcomponent 214. The various uncertainty data can be generated based onone or more of input object detection data, state data, orclassification data. For example, detection uncertainty data can bebased not only on object detection data but state data andclassification data as well. Detection uncertainty data, stateuncertainty data, and classification uncertainty data can be provided toprediction system 104. Prediction system 104 can utilize the uncertaintydata along with the object detection data, state data and classificationdata in order to generate more accurate and reliable predictionsassociated with the detected objects.

Although FIG. 3 illustrates the uncertainty data being passed fromperception system 103 to prediction system 104, the uncertainty data maybe passed within perception system 103 as well. For example, detectionuncertainty data may be passed from uncertainty system 302 into trackingcomponent 210 to be used as part of object tracking. A highly calibrateduncertainty can be generated for the output of an object detector 304.For instance, an object detector 304 may include one or moreconvolutional neural networks configured to generate object detectionsbased on a LIDAR image. An uncertainty may be calculated for an objectdetection generated from the convolutional neural network and passed totracking component 210. In turn, improved uncertainty modeling can beprovided for the output of tracking component 210. The output oftracking component 210 can be associated with uncertainty of the variousparameters of state of a detected object such as position, heading,shape, velocity, acceleration, etc.

Uncertainty system 302 may calculate uncertainty in association withobject detection data, state data, and/or classification data usingvarious techniques. For example, uncertainty system 302 may determineuncertainty for a detected object based on one or more of an objectstate, an object history, a local context associated with an object,and/or a global context associated with an object.

The state information can be used with history data associated withobject detections to determine an uncertainty with respect to whether anobject detection is an actual object. The history of an object can beexamined to choose those object detections that have a history of goodprediction and/or low uncertainty in some examples. History informationmay include one or more previous object detections that are used toassess an uncertainty for a current object detection. Additionally, ahistory of the uncertainty associated with an object can be used. Thelength of time or number of cycles associated with an object detectionand/or tracking can be used as part of a current object detection anduncertainty determination. The system may use a classification or stateassociated with an object from a previous cycle as part of determiningthe classification or state of the object for a current cycle. In someexamples, the uncertainty system may be configured to increase theuncertainty associated with an object if its classification or statechanges between cycles, as this may be indicative of a lack of certaintyin the detection or prediction.

A predetermined number of previous states of an object can be stored andused to determine uncertainty associated with the object. Each of thestored states can be passed to the prediction system. The use ofhistorical features can enable improved prediction. A predeterminednumber of future states can be forward simulated and passed as aplaceholder to be used for predicting trajectories using instance flow.Historical determinations and future predictions can be used to modifyan uncertainty associated with an object. For instance, the uncertaintyor previous state information can be modified to be consistent with afull object trajectory. By analyzing a trajectory, a determination canbe made as to what degree an object detection should be trusted bydownstream components.

A local context, such as consistency or disagreement between detectors,can be used to determine uncertainty of detected object. Uncertaintysystem 302, for example, may generate uncertainty data based on theoutputs of multiple object detectors within segmentation component 206.Rather than select the output of one object detector, multiple outputsor a fused output of each detector can be passed from the perceptionsystem. Uncertainty system 302 can calculate an uncertainty value foreach output and pass the uncertainty data with the underlying objectdetection data. In some examples, the uncertainty data may include aprobability for each of the outputs. The system may use a concurrencebetween the output of multiple pipelines to increase the uncertaintyassociated with an object detection or a disagreement between outputs todecrease the uncertainty.

A global context associated with a detected object may also be used aspart of determining uncertainty. The system may use an object's positionor locality with respect to the vehicle and/or a number of points insensor data corresponding to the object to determine an uncertaintyassociated with the object. Object detections at far range may beconsidered relatively trustworthy even when a small number of sensordata points are obtained for the object. By contrast, object detectionsat close range may be considered relatively untrustworthy when a smallnumber of sensor data points are obtained for the object. Map data canalso be used. For example, map data may indicate whether an objectdetection is consistent with a global context of an environment externalto the vehicle. In some instances, the map data is used independently ofdetection to provide an assessment of the consistency of a detectionwith the map data. Map information may be used with statisticsassociated with objects in a scene to gather information (e.g., using agraph structure) in order to provide uncertainty information associatedwith the objects.

It is noted that uncertainty system 302 may not receive all of thedepicted data in all embodiments. Uncertainty system 302 may receive oneor more of object detection data, state data, or classification data invarious examples. Additionally, uncertainty system 302 may receiveadditional data such as object association data and/or tracked objectsdata. Similarly, uncertainty system 302 may not generate uncertaintydata for all of the received outputs in some examples. Uncertaintysystem 302 may generate uncertainty data for one or more of the objectdetection data, state data, or classification data in various examples.

Although uncertainty system 302 is implemented as part of perceptionsystem 103 in FIG. 3 , uncertainty system 302 may be implemented inother manners according to various embodiments. By way of example,uncertainty system 302 may be implemented separately from perceptionsystem 103. In another example, uncertainty system may include one ormore portions implemented within one or more components of theperception system, such as within segmentation component 206, trackingcomponent 210, and/or classification component. In yet another example,uncertainty system 302 may be at least partially implemented withinprediction system 104 or another system of the vehicle computing system.

FIG. 4 is a flowchart diagram depicting an example process 400 ofgenerating and using uncertainty data within a vehicle computing systemas part of controlling an autonomous vehicle. One or more portions ofprocess 400 (and other processes described herein) can be implemented byone or more computing devices such as, for example, the computingdevices 110 within vehicle computing system 102 of FIG. 1 , or examplecomputing system 1000 of FIG. 10 . Moreover, one or more portions of theprocesses described herein can be implemented as an algorithm on thehardware components of the devices described herein (e.g., as in FIGS. 1and 10 ) to, for example, generate data indicative of uncertainty andgenerate motions plans for an autonomous vehicle using the data. Inexample embodiments, process 400 may be performed by an uncertaintysystem 302 of vehicle computing system 102.

At 402, sensor data is received from one or more sensors positionedrelative to an autonomous vehicle. By way of example, sensor data mayinclude one or more of image sensor data, RADAR sensor data, and/orLIDAR sensor data. It will be appreciated, however, that any type ofsensor data may be utilized in accordance with embodiments of thedisclosed technology.

At 404, the sensor data is input to one or more first machine-learnedmodels configured for object perception. For example, the one or morefirst machine-learned models may be object detector models provided aspart of a segmentation component of a perception system. In otherexamples, the one or more first machine-learned models may be associatedwith one or more of object tracking and/or object classification.

At 406, data indicative of one or more detected objects is obtained asan output of the one or more first machine-learned models. The data maybe one or more of object detection data, state data, or classificationdata associated with the one or more detected objects.

At 408, data indicative of at least one uncertainty is generated inassociation with the one or more object detections. The uncertainty datamay include detection uncertainty data, state uncertainty data, and/orclassification uncertainty data. The uncertainty data may be generatedusing object state information, object history information, a localcontext associated with one or more detected objects, and/or a globalcontext associated with the one or more detected objects. Theuncertainty data may be data indicative of probability associated withthe one or more object detections in some examples. By way of example,the system may generate two or more classifications for a detectedobject and assign a probability to each of the two or moreclassifications.

At 410, data indicative of the one or more object detections is input toone or more second machine-learned models configured for objectprediction. Additionally, the data indicative of the at least oneuncertainty is input to the one or more second machine-learned models.

At 412, data indicative of one or more object predictions is obtained asan output of the one or more second machine-learned models. For example,the data may include predicted state data for one or more detectedobjects, such as a predicted position, velocity, and/or trajectory of anobject. The object prediction data may be generated based at least inpart on the uncertainty input with the data indicative of the one ormore object detections. By way of example, an object predictedtrajectory may be generated based on the uncertainty associated with theclassification of the object. In another example, multiple predictedtrajectories may be generated based on multiple classifications for adetected object and probabilities passed in association with each of theclassifications.

At 414, the data indicative of one or more object predictions isprovided to a motion planning system of the autonomous vehicle. Themotion planning system can generate one or more motion plans based onone or more object predictions.

It is noted that while the uncertainty in FIG. 4 is passed to the objectprediction system as part of generating one or more object predictions,the uncertainty data may be used in other manners. By way of example,uncertainty data may be generated in association with one or more objectdetections from a segmentation component. The uncertainty data may bepassed with the object detection data to the tracking component and/orclassification component of the perception system to be used as part ofobject tracking and object classification.

FIG. 5 is a block diagram depicting an example of a vehicle computingsystem including an uncertainty system in accordance with anotherexample embodiment of the present disclosure. FIG. 5 depicts a variationof the uncertainty system 302 from FIG. 3 in which uncertainty data isprovided directly to the motion planning system 105. Additionally, theuncertainty system receives one or more outputs of the prediction system104 and generates additional prediction uncertainty data. The predictionuncertainty data can be provided from uncertainty system 302 to themotion planning system 105 along with detection uncertainty data, stateuncertainty data, and classification uncertainty data.

Prediction uncertainty data can include data indicative of one or moreuncertainties associated with predictions generated by prediction system104. By way of example, prediction uncertainty data may include anuncertainty value associated with a predicated state of a detectedobject. The uncertainty may be an uncertainty with respect to apredicted position, velocity, acceleration, trajectory, etc. associatedwith a detected object. Various uncertainty value types may be used. Forexample, the uncertainty value can be a probability associated with aprediction generated by prediction system 104. In some examples, theprobability can be a probability distribution associated with aprediction. For instance, the prediction system may generate two or moreobject predictions (e.g., multiple positions) and provide a probabilitydistribution for the multiple predictions. The motion planning system105 can then use the object predictions and probability information aspart of generating motion plans for the autonomous vehicle.

FIG. 6 is a flow chart diagram depicting an example process 500 forgenerating a motion plan based at least in part on uncertainty dataassociated with object detections in accordance with example embodimentsof the disclosed technology. In some examples, process 500 may beperformed by an uncertainty system 302 of vehicle computing system 102.

At 502, data indicative of one or more detected objects is received atthe motion planning system. The data indicative of one or more detectedobjects may include object detection data, state data, and/orclassification data.

At 504, data indicative of at least one uncertainty associated with oneor more object detections is received at the motion planning system. Thedata indicative of at least one uncertainty may include detectionuncertainty data, state uncertainty data, and/or application uncertaintydata.

At 506, data indicative of one or more object predictions is received atthe motion planning system. The data indicative of one or more objectpredictions may include predicted state data in some examples. Forexample, the predicted state data may include a predicted position, apredicted velocity, a predicted acceleration, a predicted trajectory,and/or other object prediction data.

At 508, data indicative of at least one uncertainty associated with theone or more object predictions is received at the motion planningsystem. The data indicative of at least one uncertainty associated withthe one or more object predictions may include state uncertainty data insome examples. The state uncertainty data may include an uncertaintyassociated with a predicted position, a predicted velocity, a predictedacceleration, a predicted trajectory, and/or other object predictions.

At 510, one or more motion plans are generated by the motion planningsystem. The one or more motion plans are generated based at least inpart on the uncertainty associated with the one or more objectdetections and the uncertainty associated with one or more objectpredictions. It is noted, however, that in some examples motion plansmay be generated using only uncertainty associated with one or moreobject detections or uncertainty associated with one or more objectpredictions. Moreover, in some examples, object detection data may notbe received directly by the motion planning system. Rather the objectdetection data may be processed by the prediction system which passesone or more object predictions to the motion planning system whichgenerates a motion plan based only on the output of the predictionsystem.

By way of example, uncertainty data may be used to detect gridlock orother traffic patterns associated with one or more travelways. Ratherthan requiring the use of true object detections only in suchdetermining gridlock, data indicative of object detections and dataindicative of uncertainty associate with object detections can be passeddownstream for use in informing one or more systems for identifying atraffic pattern. In this manner, such information can be used even ifthe information is not used to predict the motion of such objects.

In some examples, data associated with one or more detected objects maybe used by prediction system 104 to model continuous state predictions.This may be done using continuous values for uncertainties that areprovided from uncertainty system. Passing along such values to theprediction system may facilitate the modeling of continuous statepredictions.

In some examples, uncertainty data may be used as part of speedregression and/or defensive driving techniques for an autonomousvehicle. For example, if a perceived scene contains many uncertainobjects the motion planning system can learn or shift to discretedriving modes to be able to be more defensive around uncertain orconfusing situations. This may include slowing down around erraticpredictions, switching to another lane when uncertain pedestrians areclose to the road, switching to another lane or slowing down whenever anuncertain large vehicle is passing, when the autonomous vehicle is inthe blind spot of another vehicle, or other processes.

In some examples, the use of uncertainty data may be used in generatingmotion plans in an uncertain cost space. Uncertainty data may beassociated with object detections and then used in motion planning. Forexample using uncertainty in motion planning may allow the creation of aprobabilistic padding around objects. For example, there may be a highercost for moving toward objects that have a higher uncertainty. Such atechnique may be used where the cost space is highly non-convex. Thetechnique can be integrated as a way to only inflate the cost of thoseobjects without adding the padding in the direction of uncertainty.

FIG. 7 is a block diagram depicting an example of a vehicle computingsystem including an uncertainty system in accordance with anotherexample embodiment of the present disclosure. FIG. 7 depicts a variationin which the uncertainty system includes uncertainty generators 308,322, and 332 provided directly within various components of theperception system. Segmentation component 206 includes one or moredetector models 306. The one or more detector models can include one ormore machine-learned models configured to generate object detectiondata. One or more of the detector models 306 may include an uncertaintygenerator 308. Uncertainty generator 308 may be configured as part of amachine-learned model for object detection. In this manner, amachine-learned detector model may generate object prediction data aswell as detection uncertainty data associated with the objectdetections.

Tracking component 210 includes one or more tracking models 320. Thetracking models can include one or more machine-learned modelsconfigured to generate object state data. One or more of the trackingmodels 320 may include an uncertainty generator 322. Uncertaintygenerator 322 may be configured as part of a machine-learned model forobject tracking. In this manner, a machine-learned tracking model maygenerate object state data as well as state uncertainty data associatedwith the object state.

Classification component 214 includes one or more classification models330. The classification models can include one or more machine-learnedmodels configured to generate object classification data. One or more ofthe classification models 330 may include an uncertainty generator 332.Uncertainty generator 332 may be configured as part of a machine-learnedmodel for object classification. In this manner, a machine-learnedclassification model may generate object classification data as well asclassification uncertainty data associated with the objectclassifications.

FIG. 8 is a block diagram depicting an example of a vehicle computingsystem including an uncertainty system in accordance with anotherexample embodiment of the present disclosure. FIG. 8 depicts a variationin which the uncertainty system is implemented within one or more fusinglayers 370 of the perception system 103.

Fusing layers 370 are in communication with a plurality of perceptionpipelines 356, 358, and 360 which are each configured to process sensorA data 350, sensor B Data, and sensor N data, respectively. The one ormore fusing layers may fuse multiple perception outputs associated withdifferent sensors of the autonomous vehicle. For example, sensor A data350 may be associated with one or more first sensors, sensor B data 352may be associated with one or more second sensors, and sensor N data maybe associated with one or more third sensors. The sensor data associatedeach sensor may be processed using a perception pipeline including anindividual set of machine-learned models for object detection, objecttracking, and/or object classification. Perception pipeline A 356, forexample, may receive sensor data from the one or more first sensors andperform object segmentation, object association, tracking, and/orclassification. As such, each perception pipeline may include one ormore of the components as illustrated in FIG. 2 as part of perceptionsystem 103.

Fusing layers 370 may receive the output of each perception pipeline andgenerate detection data, state data, and/or classification data based onthe outputs of each of the perception pipelines. The fusing layers, insome instances, may select the output of a particular perceptionpipeline. For example, a first perception pipeline may assign a firstclassification to a detected object while a second perception pipelinemay assign a second classification to detect detected object. Fusinglayers 370 may implement any number of techniques for selecting amongstthe outputs of each perception pipeline. Uncertainty system 302 mayassign an uncertainty to the selected output.

In accordance with embodiments of the present disclosure, fusing layers370 may pass data indicative of one or more detected objects frommultiple perception pipelines to prediction system 104. For example,fusing layers 370 may pass a classification from perception pipeline A356 to prediction system 104 as well as another classification for thesame object from perception pipeline B 358 to prediction system 104.Uncertainty system 302 may assign an uncertainty to each of theclassifications provided to the prediction system. For example,uncertainty system 302 may generate a probability distributionassociated with multiple classifications and provide the probabilitydistribution along with the multiple classifications to the predictionsystem. Similarly, the fusing layers may in such instances, generateuncertainty data such as probability distributions for the multipleoutputs associated with the same detected object.

FIG. 9 is a flow chart diagram depicting an example process 600 forproviding a machine-learned model to generate uncertainty data inassociation with object classification of a perception system. In someexamples, process 600 may be performed by an uncertainty system 302 ofvehicle computing system 102. While process 600 is depicted with respectto object classification, it will be appreciated that similar processesmay be used to provide machine-learned models for object tracking orobject classification, for example.

At 602, one or more machine-learned models are provided that include acalibrated uncertainty for object classification. At 604, the one ormore machine-learned models are configured to generate data indicativeof an object classification. At 606, the one or more machine-learnedmodels are configured to generate a probability distribution of anobject classification.

In example embodiments, the one or more machine-learned models may be aclassifier that is configured to analyze the joint probability of allclasses given a set of input data. For example, a generative classifiercan be used in some examples. More particularly, by way of example, aGaussian naive Bayes classifier may be used. Such a classifier may makeuse of Bayes rule and a conditional independence assumption of the inputfeatures to calculate probabilities in a simple fashion. In someexamples, a Bayesian generative classifier such as a Gaussian naiveBayes classifier can be provided and have a calibrated uncertaintyvalue.

A generative classifier can produce a real probability distribution of aclass given the input data. The use of a classifier that produces a realprobability distribution may be contrasted with discriminativeclassifiers that use stochastic gradient descent (SGT) in order tominimize an arbitrary loss function. Such classifiers provide a lossfunction that is designed to push the distribution on the classprobabilities from the model to as close to possible as the groundtruth. The result of such a classifier may be to cause the distributionproduced by the classifier to be very close to either zero or onedepending on the true class. By contrast, a Bayesian generativeclassifier may analyze the underlying data and produce a true estimateof the probabilities given the distribution of inputs seen duringtraining time.

At 608, training data is provided to the one or more machine-learnedmodels. The training data may include ground truth data. The trainingdata may include labels indicating a class with which the training datais associated.

At 610, data indicative of one or more object classifications isgenerated using the one or more machine-learned models. At 612, the oneor more machine-learned models are trained to produce true estimates ofclass probabilities given a distribution of inputs. The model(s) can betrained by comparing an output of the machine-learned with theground-truth data and backpropagating errors, for example. In someexamples, training a generative classifier may be performed in a singlepass through the training data, as compared with discriminativeclassifiers that may require multiple passes to achieve optimization.The model(s) may be trained to provide an in-depth statistical analysisof the objects and may provide a way to pick the heuristic values onuncertainty while analyzing a large set of objects. In some examples,the generative classifier can relax some assumptions and learn anonlinear decision surface. In some instances, the classifier can bedetector agnostic. Because it is analyzing many conditionalprobabilities on a binary class, certain features may not show up. Thedistribution on the values of the input features can be extended to be acondition of the detection type as well as the class type

One or more machine-learned models trained to generate uncertainty datain accordance with example embodiments may provide a continuousprobability of uncertainty to the prediction system 104. Such a modelmay be able to process the state, history, local, and global informationinside of one or more fusing layers.

In some examples, Bayesian hierarchal modeling may be additionally used.For example one approach may regress values for object detection asshown in Equation 1.

p(true detection|σ_(state),σ_(history),σ_(local),σ_(global))  Equation 1

A Gaussian naïve Bayes approach may treat the parameters σ_state,σ_history, σ_local, σ_global as conditionally independent with respectto a true detection. The output can be modeled as a binomialdistribution. The relationships between these parameters can be modeledin some examples using hierarchical Bayesian methods.

Uncertainty data may be integrated with Deep prediction in someexamples. Deep prediction may make use of information such as otherobjects in a scene, map information, etc. A regression of the trueprobability distribution can be regressed directly from deep predictionwhile adding such additional information including shifting of the statedistribution. Such a technique may allow the prediction to be performedwithin perception system 103 where it may have more contextualinformation such as other detections, sensor information from thedetector, etc. This may permit reducing the amount of code and timeoverhead of integrating an additional machine-learned model

FIG. 10 depicts a block diagram of an example computing system 1000according to example embodiments of the present disclosure. The examplecomputing system 1000 includes a computing system 1002 and a machinelearning computing system 1030 that are communicatively coupled over anetwork 1080.

In some implementations, the computing system 1002 can performuncertainty data determination processes and use uncertainty data aspart of autonomous vehicle operations. In some implementations, thecomputing system 1002 can generate uncertainty data using amachine-learned model. In some implementations, the computing system1002 can be included in an autonomous vehicle. For example, thecomputing system 1002 can be on-board the autonomous vehicle. In someembodiments, computing system 1002 can be used to implement vehiclecomputing system 102. In other implementations, the computing system1002 is not located on-board the autonomous vehicle. For example, thecomputing system 1002 can operate offline to obtain sensor data andperform uncertainty data generation. The computing system 1002 caninclude one or more distinct physical computing devices.

The computing system 1002 includes one or more processors 1012 and amemory 1014. The one or more processors 1012 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory1014 can include one or more non-transitory computer-readable storagemedia, such as RAM, ROM, EEPROM, EPROM, one or more memory devices,flash memory devices, etc., and combinations thereof.

The memory 1014 can store information that can be accessed by the one ormore processors 1012. For instance, the memory 1014 (e.g., one or morenon-transitory computer-readable storage mediums, memory devices) canstore data 1016 that can be obtained, received, accessed, written,manipulated, created, and/or stored. The data 1016 can include, forinstance, image or other sensor data captured by one or more sensors,machine-learned models, etc. as described herein. In someimplementations, the computing system 1002 can obtain data from one ormore memory device(s) that are remote from the computing system 1002.

The memory 1014 can also store computer-readable instructions 1018 thatcan be executed by the one or more processors 1012. The instructions1018 can be software written in any suitable programming language or canbe implemented in hardware. Additionally, or alternatively, theinstructions 1018 can be executed in logically and/or virtually separatethreads on processor(s) 1012.

For example, the memory 1014 can store instructions 1018 that whenexecuted by the one or more processors 1012 cause the one or moreprocessors 1012 to perform any of the operations and/or functionsdescribed herein, including, for example, generating machine-learnedmodels, generating uncertainty data, etc.

According to an aspect of the present disclosure, the computing system1002 can store or include one or more machine-learned models 1010. Asexamples, the machine-learned models 1010 can be or can otherwiseinclude various machine-learned models such as, for example, neuralnetworks (e.g., deep neural networks or other types of models includinglinear models and/or non-linear models. Example neural networks includefeed-forward neural networks, recurrent neural networks (e.g., longshort-term memory recurrent neural networks), convolutional neuralnetworks, or other forms of neural networks.

In some implementations, the computing system 1002 can receive the oneor more machine-learned models 1010 from the machine learning computingsystem 1030 over network 1080 and can store the one or moremachine-learned models 1010 in the memory 1014. The computing system1002 can then use or otherwise implement the one or more machine-learnedmodels 1010 (e.g., by processor(s) 1012). In particular, the computingsystem 1002 can implement the machine-learned model(s) 1010 to generateuncertainty data for object detections, predictions, and motion plangeneration based on sensor data.

The machine learning computing system 1030 includes one or moreprocessors 1032 and a memory 1034. The one or more processors 1032 canbe any suitable processing device (e.g., a processor core, amicroprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.)and can be one processor or a plurality of processors that areoperatively connected. The memory 1034 can include one or morenon-transitory computer-readable storage media, such as RAM, ROM,EEPROM, EPROM, one or more memory devices, flash memory devices, etc.,and combinations thereof. In some embodiments, machine learningcomputing system 1030 can be used to implement vehicle computing system102.

The memory 1034 can store information that can be accessed by the one ormore processors 1032. For instance, the memory 1034 (e.g., one or morenon-transitory computer-readable storage mediums, memory devices) canstore data 1036 that can be obtained, received, accessed, written,manipulated, created, and/or stored. The data 1036 can include, forinstance, machine-learned models and flow graphs as described herein. Insome implementations, the machine learning computing system 1030 canobtain data from one or more memory device(s) that are remote from themachine learning computing system 1030.

The memory 1034 can also store computer-readable instructions 1038 thatcan be executed by the one or more processors 1032. The instructions1038 can be software written in any suitable programming language or canbe implemented in hardware. Additionally, or alternatively, theinstructions 1038 can be executed in logically and/or virtually separatethreads on processor(s) 1032.

For example, the memory 1034 can store instructions 1038 that whenexecuted by the one or more processors 1032 cause the one or moreprocessors 1032 to perform any of the operations and/or functionsdescribed herein, including, for example, generating uncertainty dataand controlling an autonomous vehicle based on data indicative ofuncertainty associated with detection objects external to the autonomousvehicle.

In some implementations, the machine learning computing system 1030includes one or more server computing devices. If the machine learningcomputing system 1030 includes multiple server computing devices, suchserver computing devices can operate according to various computingarchitectures, including, for example, sequential computingarchitectures, parallel computing architectures, or some combinationthereof.

In addition or alternatively to the machine-learned model(s) 1010 at thecomputing system 1002, the machine learning computing system 1030 caninclude one or more machine-learned models 1040. As examples, themachine-learned models 1040 can be or can otherwise include variousmachine-learned models such as, for example, neural networks (e.g., deepneural networks) or other types of models including linear models and/ornon-linear models. Example neural networks include feed-forward neuralnetworks, recurrent neural networks (e.g., long short-term memoryrecurrent neural networks), convolutional neural networks, or otherforms of neural networks.

As an example, the machine learning computing system 1030 cancommunicate with the computing system 1002 according to a client-serverrelationship. For example, the machine learning computing system 1030can implement the machine-learned models 1040 to provide a web serviceto the computing system 1002. For example, the web service can generateuncertainty data in response to sensor data and/or other data receivedfrom an autonomous vehicle.

Thus, machine-learned models 1010 can located and used at the computingsystem 1002 and/or machine-learned models 1040 can be located and usedat the machine learning computing system 1030.

In some implementations, the machine learning computing system 1030and/or the computing system 1002 can train the machine-learned models1010 and/or 1040 through use of a model trainer 1060. The model trainer1060 can train the machine-learned models 1010 and/or 1040 using one ormore training or learning algorithms. One example training technique isbackwards propagation of errors. In some implementations, the modeltrainer 1060 can perform supervised training techniques using a set oflabeled training data. In other implementations, the model trainer 1060can perform unsupervised training techniques using a set of unlabeledtraining data. The model trainer 1060 can perform a number ofgeneralization techniques to improve the generalization capability ofthe models being trained. Generalization techniques include weightdecays, dropouts, or other techniques.

In particular, the model trainer 1060 can train a machine-learned model1010 and/or 1040 based on a set of training data 1062. The training data1062 can include, for example, ground truth data including annotationsfor sensor data portions and/or vehicle state data. The model trainer1060 can be implemented in hardware, firmware, and/or softwarecontrolling one or more processors.

In some examples, the model trainer 160 can train a machine-learnedmodel 1010 and/or 1040 configured to generate uncertainty data inassociation with detected objects. In some examples, the machine-learnedmodel 1010 and/or 1040 is trained using sensor data that has beenlabeled or otherwise annotated as having a correspondence to a detectedobject, a class of a detected object, etc. By way of example, sensordata collected in association with a particular class of object can belabeled to indicate that it corresponds to an object detection or theparticular class. In some instances, the label may be a simpleannotation that the sensor data corresponds to a positive trainingdataset.

The computing system 1002 can also include a network interface 1024 usedto communicate with one or more systems or devices, including systems ordevices that are remotely located from the computing system 1002. Thenetwork interface 1024 can include any circuits, components, software,etc. for communicating with one or more networks (e.g., 1080). In someimplementations, the network interface 1024 can include, for example,one or more of a communications controller, receiver, transceiver,transmitter, port, conductors, software and/or hardware forcommunicating data. Similarly, the machine learning computing system1030 can include a network interface 1064.

The network(s) 1080 can be any type of network or combination ofnetworks that allows for communication between devices. In someembodiments, the network(s) can include one or more of a local areanetwork, wide area network, the Internet, secure network, cellularnetwork, mesh network, peer-to-peer communication link and/or somecombination thereof and can include any number of wired or wirelesslinks. Communication over the network(s) 1080 can be accomplished, forinstance, via a network interface using any type of protocol, protectionscheme, encoding, format, packaging, etc.

FIG. 10 illustrates one example computing system 1000 that can be usedto implement the present disclosure. Other computing systems can be usedas well. For example, in some implementations, the computing system 1002can include the model trainer 1060 and the training data 1062. In suchimplementations, the machine-learned models 1010 can be both trained andused locally at the computing system 1002. As another example, in someimplementations, the computing system 1002 is not connected to othercomputing systems.

In addition, components illustrated and/or discussed as being includedin one of the computing systems 1002 or 1030 can instead be included inanother of the computing systems 1002 or 1030. Such configurations canbe implemented without deviating from the scope of the presentdisclosure. The use of computer-based systems allows for a great varietyof possible configurations, combinations, and divisions of tasks andfunctionality between and among components. Computer-implementedoperations can be performed on a single component or across multiplecomponents. Computer-implemented tasks and/or operations can beperformed sequentially or in parallel. Data and instructions can bestored in a single memory device or across multiple memory devices.

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations and/or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. For instance, features illustrated or described aspart of one embodiment can be used with another embodiment to yield astill further embodiment. Thus, it is intended that the presentdisclosure cover such alterations, variations, and equivalents.

1.-20. (canceled)
 21. A computer-implemented method, comprising:determining, by a perception system of an autonomous vehicle, dataindicative of an uncertainty associated with a detected object in anenvironment of the autonomous vehicle at a current cycle, wherein thedata indicative of the uncertainty is determined based on dataindicative of the detected object and history data associated with thedetected object at a previous cycle; providing the data indicative ofthe detected object and the data indicative of the uncertaintyassociated with the detected object to one or more machine-learnedmodels within a motion planning system of the autonomous vehicle; andgenerating, by the motion planning system, a motion plan for theautonomous vehicle based at least in part on the uncertainty.
 22. Thecomputer-implemented method of claim 21, wherein the data indicative ofthe uncertainty indicates at least one probability distribution withrespect to the detected object.
 23. The computer-implemented method ofclaim 21, wherein the data indicative of the uncertainty indicates anuncertainty with respect to a detection of the detected object.
 24. Thecomputer-implemented method of claim 21, wherein the data indicative ofthe uncertainty indicates an uncertainty with respect to aclassification of the detected object.
 25. The computer-implementedmethod of claim 21, wherein: the data indicative of the detected objectincludes data indicative of a classification of the detected object; andthe data indicative of the uncertainty includes a probabilitydistribution associated with the classification.
 26. Thecomputer-implemented method of claim 21, wherein the data indicative ofthe uncertainty indicates an uncertainty with respect to state dataindicative of a state of the detected object.
 27. Thecomputer-implemented method of claim 21, wherein the history datacomprises information associated with one or more detections of thedetected object at the previous cycle.
 28. The computer-implementedmethod of claim 21, wherein the history data comprises a length of timeduring which the detected object has been tracked.
 29. Thecomputer-implemented method of claim 21, wherein the history datacomprises a number of cycles during which the detected object has beentracked.
 30. The computer-implemented method of claim 21, wherein thehistory data comprises at least one of a classification or a state ofthe detected object from the previous cycle.
 31. An autonomous vehiclecontrol system for controlling an autonomous vehicle, the autonomousvehicle control system comprising: one or more processors; and one ormore tangible, non-transitory, computer-readable media that storeinstructions that are executable to cause the autonomous vehicle controlsystem to perform operations, the operations comprising: determining, bya perception system of the autonomous vehicle control system, dataindicative of an uncertainty associated with a detected object in anenvironment of the autonomous vehicle at a current cycle, wherein thedata indicative of the uncertainty is determined based on dataindicative of the detected object and history data associated with thedetected object at a previous cycle; providing the data indicative ofthe detected object and the data indicative of the uncertaintyassociated with the detected object to one or more machine-learnedmodels within a motion planning system of the autonomous vehicle controlsystem; and generating, by the motion planning system, a motion plan forthe autonomous vehicle based at least in part on the uncertainty. 32.The autonomous vehicle control system of claim 31, wherein the dataindicative of the uncertainty indicates at least one probabilitydistribution with respect to the detected object.
 33. The autonomousvehicle control system of claim 31, wherein the data indicative of theuncertainty indicates an uncertainty with respect to at least one of adetection of the detected object or a classification of the detectedobject.
 34. The autonomous vehicle control system of claim 31, whereinthe data indicative of the uncertainty indicates an uncertainty withrespect to state data indicative of a state of the detected object. 35.The autonomous vehicle control system of claim 31, wherein the historydata comprises information associated with one or more detections of thedetected object at the previous cycle.
 36. The autonomous vehiclecontrol system of claim 31, wherein the history data comprises a lengthof time during which the detected object has been tracked.
 37. Theautonomous vehicle control system of claim 31, wherein the history datacomprises a number of cycles during which the detected object has beentracked.
 38. The autonomous vehicle control system of claim 31, whereinthe history data comprises at least one of a classification or a stateof the detected object from the previous cycle.
 39. An autonomousvehicle, comprising: one or more processors; and one or more tangible,non-transitory, computer-readable media that store instructions that areexecutable to cause the one or more processors to perform operations,the operations comprising: determining, by a perception system of theautonomous vehicle, data indicative of an uncertainty associated with adetected object in an environment of the autonomous vehicle at a currentcycle, wherein the data indicative of the uncertainty is determinedbased on data indicative of the detected object and history dataassociated with the detected object at a previous cycle; providing thedata indicative of the detected object and the data indicative of theuncertainty associated with the detected object to one or moremachine-learned models within a motion planning system of the autonomousvehicle; and generating, by the motion planning system, a motion planfor the autonomous vehicle based at least in part on the uncertainty.40. The autonomous vehicle of claim 39, wherein the history datacomprises at least one of a length of time or a number of cycles duringwhich the detected object has been tracked.